Very cool ty! I'm not as familiar with emscripten, I'll take some time to look at this and whether there is a way to support it gracefully.

@ggerganov one thing I really like is the tokenizer inside C here, removing the need to use the run_wrap.py and sentencepiece. But it's not clear how you produced vocab.bin, presumably it's an export script?

@ggerganov one thing I really like is the tokenizer inside C here, removing the need to use the run_wrap.py and sentencepiece. But it's not clear how you produced vocab.bin, presumably it's an export script?

nvm got it working in 3bfa566

Yes, for this PR I had exported the vocab from llama.cpp, which has been generated with a similar script as yours.

I'm not sure about the cause of the leading space - we have it in llama.cpp output as well and haven't dug to understand how to fix it yet.

The tokenizer stores a leading space on some tokens.

In python we use the Decode function, that removes the leading space on the first token

But this repo is using the id_to_piece() function to export the tokenizer

python3
>>> import sentencepiece as spm
>>> sp = spm.SentencePieceProcessor(model_file='tokenizer.model')
>>> sp.id_to_piece([9038])
['▁Once']

Then it is simply printed, without considering if it is the first token:

The PR #89 fixes it

I recreated @ggerganov's changes against the current tree (with support for prompt): https://github.com/gohai/llama2.c/commits/emscripten

I've built a JS framework around this idea to run Language Models on Web.
This also leverages Emscripten to compile llama2.c into WASM JS.

llama2.c on Web Demo: https://rahuldshetty.github.io/ggml.js-examples/llama2_tinystories.html
ggml.js Framework: https://github.com/rahuldshetty/ggml.js